Repository navigation
Never resend a non-poll request to recover server capabilities - #306
Conversation
#304 resent any request once with "none" when its answer named an unknown capabilities digest. A completion's answer can carry the next leased task, and a resent completion only gets a duplicate receipt without it, so a worker could lose leased work. Only task polls (idempotent by poll request ID) are still resent. For any other request the answer is kept as received, the full capabilities are read with GET /api/cluster/info (worker_protocol.server_capabilities) and substituted for the unknown digest, and the held pair is forgotten so later requests send none. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review fixes: - After a garbled digest the one cluster-info read used `?`, so a failed read threw away an answer already received (including its next task) and stopped the worker. The answer is now used without the capabilities (the unknown copies are dropped), the held digest is forgotten, and later requests learn them again with `none`. - Any cluster-info failure used to stop a running worker. Startup discovery (query-task authority, Nexus cooperation) and the inline payload policy read before a report now retry a connection failure, timeout, 5xx or 429 with the shared jittered backoff and the 60 s / 5 min ERROR escalation until they succeed or the worker stops. Refusals still return; Clients outside a running worker make one attempt. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review fixes: - When the 60 s payload-policy cache expired during a cluster-info outage, every report waited for cluster info. An expired policy is now refreshed with one attempt; on an outage the last-known policy is used and discovery is tried again 10 s later. Only a worker with no policy yet waits under the outage rules. - An activity report sent inline under a stale policy, after the Server lowered its inline limit, was refused with 413 payload_too_large and lost. The worker now forgets the policy, rediscovers it (outage rules apply), encodes the same report again, externalizing what the new limit requires, and resends it once. A second refusal keeps today's handling. Workflow-task completions keep their typed size refusals. Co-Authored-By: Claude Opus 5.5 <[email protected]>
A completion still refused as too large after the payload policy was rediscovered, or one that could not be externalized (above the upload limit, or no payload storage), returned an error from the settlement, and the poll lane treated it as fatal: the worker exited. Such an attempt is now reported as a non-retryable ActivityResultTooLarge failure (failure.type and failure.class), as the PHP SDK does, and the worker keeps running. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
rmcdaniel
left a comment
There was a problem hiding this comment.
Reviewed the coordinated worker payload-limit fix against the PHP #451, Python #360 and Rust #306 changes. PHP, Python and Rust recover only from the Server's typed size refusal. The worker can rediscover the policy and re-encode the result without re-running the activity. A result that still cannot fit is reported with type and class ActivityResultTooLarge and non_retryable: true, and worker dispatch remains available. A discovery outage keeps the attempt eligible for recovery instead of recording a permanent size failure.
The target-branch and language CI jobs passed for all three reviewed heads. Python also passed the local focused transport suite, and Rust passed the 17 worker-outage tests. The release qualification and published-package checks follow the merges.



Follow-up to #304, applying the cross-SDK principle: never resend a non-idempotent request to recover server capabilities.
Problem
In #304, when an answer named an unknown capabilities digest, the SDK resent that request once with
none. That covered every request kind, including completions, failures, heartbeats, starts and signals. A workflow-task completion's answer can carry the next leased task, and resending the completion only returns a duplicate receipt without that task. The PHP SDK lost leased activities exactly this way.Change (
send_json_with_timeout,src/server_capabilities_digest.rs)Task polls (
/worker/{workflow,activity,query}-tasks/poll) are idempotent bypoll_request_id. They are still resent once withnone, as before.Every other request is never resent:
GET /api/cluster/info, fromworker_protocol.server_capabilities(the same object);Digest::restore_withsubstitutes that object for the unknown digest string, at the same depth rules as before;none.The received answer is never failed. If that single cluster-info read fails (a 5xx, or a refused or dropped connection), the answer, including any next task, is still used:
noneand learn the capabilities again.Cluster info never recurses into itself.
Cluster-info outages no longer stop a running worker. This covers startup discovery (query-task authority, Nexus cooperation) and the inline-payload policy read before a report.
uniform(0, min(4 s, 0.25 s·2^(n−1))).tracingERROR after 60 s, then every 5 minutes, and resets on success.retryable: false, 401, 403 and other 4xx) still return.Client::cluster_info, and any Client outside a running worker, makes one attempt.CHANGELOG (Unreleased) and the README are updated.
API: no public change.
cargo semver-checks --baseline-version 4.21.0reports229 checks: 229 pass, 31 skipandSummary no semver update required.Stale payload policy during an outage. The 60 s payload-policy cache used to make reports wait for cluster info once it expired during an outage. Now an expired policy is refreshed with one attempt. On an outage the last-known policy is used and discovery is retried 10 s later, as in PHP #451. Only a worker with no policy yet waits under the outage rules.
Too-large report after a lowered limit. An activity report sent inline under a stale policy, then refused with 413
payload_too_large, is no longer lost:Workflow-task completions keep their typed size refusals (
managed_worker_leaves_other_413_and_5xx_completion_refusals_unchanged).Still too large: fail the attempt, not the worker. Before this change, a result that was refused again after rediscovery, or that could not be externalized (above the upload limit, or no storage), made the settlement return an error that the poll lane treated as fatal, and the worker exited. Such an attempt is now reported as a non-retryable
ActivityResultTooLargefailure (failure.typeandfailure.class), as in PHP, and the worker keeps running.Tests (
tests/server_capabilities_digest.rs,src/tests/worker_transient_outage.rs)a_report_refused_after_a_lowered_limit_is_externalized_and_resent_oncea_result_still_too_large_after_rediscovery_fails_the_attempt_not_the_workerActivityResultTooLargefail is sent, and the poll lane keeps running.an_expired_policy_keeps_reports_moving_through_a_cluster_info_outagea_garbled_completion_answer_is_kept_and_capabilities_come_from_cluster_inforecorded,next_task) is returned, withserver_capabilitiesrestored fromGET /api/cluster/info. The cluster-info request and the next poll both sendnone.a_garbled_answer_survives_unavailable_cluster_infonone.startup_cluster_info_failures_are_waited_outrun_time_cluster_info_failures_are_waited_outcluster_info_failures_other_than_refusals_are_transientworker_answers_are_restored_from_the_digest_each_request_sentnone, and a second unknown digest isError::Protocol.Results:
cargo fmt --check: clean.cargo clippy --all-targets --all-features -- -D warnings: clean.cargo test: 1120 passed, 0 failed, 82 ignored (server-gated).cargo +1.86.0 check --all-targets: passes.🤖 Generated with Claude Code